Pyramid In Memory Database

Pyramid's in memory database (IMDB) can be used as a datasource, allowing you to build a data flow based on data models that were previously built in Pyramid and loaded into an IMDB target.

The process for connecting to IMDB as a source is the same as that for connecting to relational databases.

• Click here to learn how to connect to IMDB as a target.

Connect to Pyramid IMDB as a Source

Add the In Memory node to the canvas and go to its Properties panel, where you need to connect to the source (red highlight below) and determine which tables to add to the data flow (green highlight). You can also add notes in the Description window (blue highlight).

Select the required server (green highlight below). If the server doesn't appear in the list, try clicking refresh (blue arrow below). Admins can also configure additional servers by clicking the Add Server button (orange arrow).

Next, select the required database (red highlight). If the database isn't listed, try refreshing the list (purple arrow).

To enable direct querying, select 'Direct Query Datasource', and then progress immediately to Data Modeling. However, if you want to create a flow diagram and apply data cleansing, or simply don't want to allow direct querying of the model, do not enable direct querying.

Table Selection

Go to the Tables window (image below) to choose which tables to copy into the new data model. Table selection is relevant for both direct querying and data ingestion. Click the refresh button (green arrow below) to ensure the list of tables is up to date, and use the 'Filter Table List' field to search for tables.

Add Tables to the Data Flow

Once you've selected the required tables, you'll need to add them to the data flow (unless you've enabled direct query).

If you want to apply data cleansing, manipulation, or machine learning to the model, copy the selected tables by clicking the 'Add Tables' button (yellow highlight). Each selected table will be copied to an individual table node, to which you connect a range of functions and formulations.

If you don't intend to apply any data cleansing, you can copy them sing the 'Add as Multi-Select' button (blue highlight). This latter option copies all selected tables to a single node, using the multi-select function. The resulting node must then be connected directly to the target.

If you have enabled direct query, the 'Add' buttons will be disabled, as no nodes can be connected to a source designated for direct query.

Another way to add tables from the source to the data flow is via the Select functions, using the single-select Table or multi-select Tables nodes. You can then input the column(s) for each select operation. Another option is to use the Query node to copy a data set from the source using an SQL or SOQL expression.

Description

You can add text to the Description window, which is useful for documenting the ETL pipeline.